Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment: Analyzing the Sentiment Bias of Four Major Tools

Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these are, and whether the ratings given by one tool agree with those another tool. We use two datasets - (1) NEWS consisting 5,880 news stories 60K comments from four social media platforms: Twitter, Instagram, YouTube, Facebook; (2) IMDB 7,500 positive negative movie reviews investigate agreement bias widely used (SA) tools: Microsoft Azure (MS), IBM Watson, Google Cloud, Amazon Web Services (AWS). find that assign same on less than half (48.1%) analyzed content. also AWS exhibits neutrality both datasets, bi-polarity dataset but dataset, MS exhibit no clear have dataset. Overall, has highest accuracy relative ground truth Findings indicate psycholinguistic features especially affect, tone, adjectives explain why disagree. Engineers urged caution when implementing SA for applications, as selection affects obtained labels.